Systems and methods for prioritizing object prediction for autonomous vehicles

ABSTRACT

Systems and methods for determining object prioritization and predicting future object locations for an autonomous vehicle are provided. A method can include obtaining, by a computing system comprising one or more processors, state data descriptive of at least a current or past state of a plurality of objects that are perceived by an autonomous vehicle. The method can further include determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object. The method can further include determining, by the computing system, an order at which the computing system determines a predicted future state for each object based at least in part on the priority classification for each object and determining, by the computing system, the predicted future state for each object based at least in part on the determined order.

PRIORITY CLAIM

The present application is a continuation of and claims priority to U.S.patent application Ser. No. 16/211,376 having a filing date of Dec. 6,2018 and U.S. patent application Ser. No. 15/811,865 having a filingdate of Nov. 14, 2017, both of which are based on and claim priority toU.S. Provisional Application 62/549,407 having a filing date of Aug. 23,2017. U.S. patent application Ser. No. 16/211,376, U.S. patentapplication Ser. No. 15/811,865, and U.S. Provisional Application62/549,407 are hereby incorporated by reference herein in theirentirety.

FIELD

The present disclosure relates generally to autonomous vehicles. Moreparticularly, the present disclosure relates to systems and methods fordetermining a priority classification for objects that are perceived byautonomous vehicles and predicting a future location for the objectsbased at least in part on the respective priority classification foreach object.

BACKGROUND

An autonomous vehicle is a vehicle that is capable of sensing itsenvironment and navigating with minimal or no human input. Inparticular, an autonomous vehicle can observe its surroundingenvironment using a variety of sensors and can attempt to comprehend theenvironment by performing various processing techniques on datacollected by the sensors. Given knowledge of its surroundingenvironment, the autonomous vehicle can identify an appropriate motionpath through such surrounding environment.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to acomputer-implemented method. The method can include obtaining, by acomputing system comprising one or more processors, state datadescriptive of at least a current or past state of a plurality ofobjects that are perceived by an autonomous vehicle. The method canfurther include determining, by the computing system, a priorityclassification for each object in the plurality of objects based atleast in part on the respective state data for each object. The methodcan further include determining, by the computing system, an order atwhich the computing system determines a predicted future state for eachobject based at least in part on the priority classification for eachobject. The method can further include determining, by the computingsystem, the predicted future state for each object based at least inpart on the determined order.

Another example aspect of the present disclosure is directed to acomputing system. The computing system can include a perception systemcomprising one or more processors. The perception system can beconfigured to generate, for each of a plurality of consecutive timeframes, state data descriptive of at least a current state of each of aplurality of objects that are perceived by an autonomous vehicle. Thecomputing system can further include a priority classification systemcomprising one or more processors. The priority classification systemcan be configured to, for each of the plurality of consecutive timeframes, classify each object in the plurality of objects as eitherhigh-priority or low-priority based at least in part on the respectivestate data for each object. The computing system can further include aprediction system comprising one or more processors. The predictionsystem can be configured to, for each of the plurality of consecutivetime frames receive the priority classification for each respectiveobject, determine, for the current time frame, a predicted future statefor each object classified as high-priority, and provide the predictedfuture state for each object classified as high-priority for the currenttime frame to a motion planning system implemented by the one or moreprocessors.

Another example aspect of the present disclosure is directed to anautonomous vehicle. The autonomous vehicle can include one or moreprocessors and one or more non-transitory computer-readable media thatcollectively store instructions that, when executed by the one or moreprocessors, cause the one or more processors to perform operations. Theoperations can include obtaining state data descriptive of at least acurrent or past state of a plurality of objects that are perceived bythe autonomous vehicle. The operations can further include determining apriority classification for each object in the plurality of objectsbased at least in part on the respective state data for each object. Theoperations can further include determining an order at which thecomputing system determines a predicted future state for each objectbased at least in part on the priority classification for each object.The operations can further include determining the predicted futurestate for each object based at least in part on the determined order.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, user interfaces,and electronic devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example autonomous vehicleaccording to example aspects of the present disclosure;

FIG. 2 depicts an example perception system according to example aspectsof the present disclosure;

FIG. 3 depicts an example prediction system according to example aspectsof the present disclosure;

FIG. 4 depicts a block diagram of an example object prediction processaccording to example aspects of the present disclosure;

FIG. 5 depicts a block diagram of an example computing system accordingto example aspects of the present disclosure; and

FIG. 6 depicts a flow chart diagram of an example method to determine amotion plan for an autonomous vehicle according to example aspects ofthe present disclosure.

DETAILED DESCRIPTION

Generally, the present disclosure is directed to systems and methods fordetermining a priority classification for objects that are perceived byautonomous vehicles and predicting a future location for the objectsbased at least in part on the respective priority classification foreach object. In particular, an autonomous vehicle can include orotherwise use a prediction system to predict the future locations of theobjects such as, for example, other vehicles, pedestrians, bicyclists,etc. based at least in part on perception information that describescurrent and/or past states of the objects and/or the surroundingenvironment. In some implementations, the autonomous vehicle can includeor otherwise use a priority classification system to classify arespective priority of each object perceived by the perception system.For example, in some implementations, each object can be classified aseither high-priority or low-priority. The prediction system candetermine a predicted future state for each object based at least inpart on the priority classification for each object. For example, insome implementations, the order at which the computing system determinesthe predicted future state for each object can be determined based onthe priority classification for each object, such as, for example, bydetermining the predicted future state for objects classified ashigh-priority before predicted future states are determined for objectsclassified as low-priority.

As one example, in a system that operates iteratively over a number oftime frames, the prediction system can determine in a current time framethe predicted future state for each object classified as high-priorityin the current time frame. However, the prediction system can wait anddetermine (e.g., in a subsequent time frame or at least subsequent toproviding the predicted future states for each object classified ashigh-priority to a motion planning system) the predicted future statefor each object classified as low-priority in the current time frame. Insuch fashion, predicted future states for high-priority objects can bedelivered to the motion planning system in an advanced fashion (e.g.,“ahead of the schedule”), thereby allowing the motion planning systemadditional time to determine a motion plan relative to the high-priorityobjects and the vehicle additional time to implement the determinedmotion plan. In such fashion, the autonomous vehicle can be controlledto react more quickly relative to objects classified as high-priority.For example, the additional time gained through advancing the predictedfuture states by the prediction system can enable to vehicle to come toa stop more quickly or otherwise make improved maneuvers which enhancepassenger and vehicle safety.

According to another aspect of the present disclosure, in someimplementations, the type of prediction system used for determining thepredicted future state for each object can be determined based on thepriority classification for each object. For example, in someimplementations a high-fidelity prediction system can be used forobjects classified as high-priority, whereas a low-fidelity predictionsystem can be used for objects classified as low-priority.

According to another aspect of the present disclosure, the priorityclassification system described herein can include or leverage one ormore machine-learned models that assist in classifying each objectperceived by the autonomous vehicle. As an example, in someimplementations, the priority classification system can include amachine-learned object classifier configured to classify each perceivedobject, such as by classifying each object as high-priority orlow-priority. The use of machine-learned models can improve the speed,quality, and/or accuracy of object priority classification. The improvedability to classify objects according to priority can allow for moreefficient use of prediction system resources by, for example, allowingfor future states of higher priority objects to be predicted beforelower priority objects. Further, this can allow for the predicted futurestates for higher priority objects to be provided to a motion planningsystem sooner, reducing overall latency for determining a motion plan,thereby reducing autonomous vehicle response times and enhancingpassenger safety and vehicle efficiency.

More particularly, in some implementations, an autonomous vehicle caninclude a computing system that assists in controlling the autonomousvehicle. The autonomous vehicle can be a ground-based autonomous vehicle(e.g., car, truck, bus, etc.), an air-based autonomous vehicle (e.g.,airplane, drone, helicopter, or other aircraft), or other types ofvehicles (e.g., watercraft). In some implementations, the computingsystem can include a perception system, a prediction system, and amotion planning system that cooperate to perceive the surroundingenvironment of the autonomous vehicle and determine a motion plan forcontrolling the motion of the autonomous vehicle accordingly. Forexample, the perception system can perceive one or more objects that areproximate to an autonomous vehicle, and provide state data indicative ofthe one or more objects to the prediction system. The prediction systemcan then determine a predicted future state for each object perceived bythe perception system. The motion planning system can then determine amotion plan for the autonomous vehicle based on the predicted futurestates for the objects. In this way, an autonomous vehicle can perceiveobjects proximate to the autonomous vehicle, and, in response, controlthe autonomous vehicle accordingly.

In some implementations, an autonomous vehicle can perform each of theperception, prediction, and motion planning steps sequentially usingdata obtained in a plurality of consecutive time frames. For example,for a time frame N, the perception system can receive sensor data forthe time frame N; the perception system can concurrently generate andprovide state data to the prediction system for one or more objectsperceived by the perception system for a time frame N−1; the predictionsystem can concurrently determine a predicted future state for eachobject perceived by the perception system for a time frame N−2; and amotion planning system can concurrently determine a motion plan for theautonomous vehicle using predicted future states for a time frame N−3.Thus, a motion plan for the autonomous vehicle can be iterativelydetermined using data from each of a plurality of consecutive timeframes.

However, in such an implementation, each of the perception, prediction,and motion planning systems may require the preceding system to completea respective analysis of data for a time frame before each system cansubsequently analyze the data for the time frame. For example, for eachconsecutive time frame, the perception system may need to complete ananalysis of data obtained from one or more sensors on the autonomousvehicle in order to generate the state data used by the predictionsystem. Similarly, the prediction system may need to complete ananalysis of state data for a time frame to determine a predicted futurestate for each object before the motion planning system can determine amotion plan for the autonomous vehicle. Thus, the overall time from whenan object is sensed by a sensor until a motion plan is determined inresponse to the object may be dependent upon each system completing itsrespective analysis of the object along with all other objects perceivedat the same time as the object.

In contrast, the systems and methods according to example aspects of thepresent disclosure can allow for determining a priority classificationfor objects perceived by an autonomous vehicle and determining apredicted future state for each object based at least in part on thepriority classification for each object, thereby enabling higherpriority objects to be analyzed before lower priority objects.

In particular, in some implementations, the perception system canreceive sensor data from one or more sensors that are coupled to orotherwise included within the autonomous vehicle. As examples, the oneor more sensors can include a Light Detection and Ranging (LIDAR)system, a Radio Detection and Ranging (RADAR) system, one or morecameras (e.g., visible spectrum cameras, infrared cameras, etc.), and/orother sensors. The sensor data can include information that describesthe location of objects within the surrounding environment of theautonomous vehicle. In some implementations, the sensor data can beobtained at a plurality of consecutive time frames. Based on sensor datareceived from the one or more sensors and/or the map data, theperception system can identify one or more objects that are proximate tothe autonomous vehicle at each time frame. As an example, in someimplementations, the perception system can segment the sensor data(e.g., the LIDAR data) into discrete object polygons and/or trackobjects frame-to-frame (e.g., iteratively over a number of consecutivetime frames or periods).

In particular, in some implementations, the perception system cangenerate, for each object, state data that describes a current state ofsuch object (also referred to as one or more features of the object). Asexamples, the state data for each object can describe an estimate of theobject's: location (also referred to as position); speed (also referredto as velocity); acceleration; heading; yaw rate; orientation;size/footprint (e.g., as represented by a bounding polygon or othershape); type/class (e.g., vehicle, pedestrian, bicycle); distance fromthe autonomous vehicle; minimum path to interaction with the autonomousvehicle; a minimum time duration to interaction with the autonomousvehicle; and/or other state information and/or covariances of theabove-described forms of state information. In some implementations,certain state data for an object can be used to determine one or moreother features for the object. For example, in some implementations, anobject's position, speed, acceleration, and/or heading can be used todetermine a minimum path to interaction with the autonomous vehicle or aminimum time duration to interaction with the autonomous vehicle. Theperception system can provide the state data to the priorityclassification system and/or the prediction system (e.g., iterativelyfor each time frame).

According to an aspect of the present disclosure, the autonomous vehiclecan further include a priority classification system configured toclassify each object perceived by the autonomous vehicle. In someimplementations, the priority classification system can be included inor otherwise incorporated into the perception system. In someimplementations, the priority classification system can be included inor otherwise incorporated into the prediction system. The priorityclassification system can classify objects perceived by the perceptionsystem based on the state data for each object. For example, thepriority classification system can classify each object into one of aplurality of priority categories and/or rank each object relative toeach other object. The relative priority classification and/or rank foreach object can be determined based on the state data for each object.The priority classification for each object can be indicative of animportance of the object to a determination for a motion plan for theautonomous vehicle. As examples, the priority classification assigned toeach object can be based on a plurality of factors, such as how likelyan object is to interact with the autonomous vehicle, how soon an objectis likely to interact with the autonomous vehicle, whether an object islikely to impact a motion plan for the autonomous vehicle, etc. Forexample, a vehicle traveling at a high rate of speed towards theautonomous vehicle can be classified as a higher priority object than avehicle traveling away from the autonomous vehicle.

In some implementations, the priority classification can be based on oneor more heuristic processes. For example, one or more thresholds can beused to classify objects based on one or more features of the object.For example, a minimum time duration, a minimum path, or a minimumdistance to interaction with the autonomous vehicle can be used toclassify the objects based on how far away the objects are from theautonomous vehicle or how soon the objects will likely interact with theautonomous vehicle. Similarly, a heading and/or velocity can be used toclassify objects. For example objects traveling on headings away fromthe autonomous vehicle can be classified as lower priority than objectstraveling towards the autonomous vehicle, and objects traveling athigher speeds towards the autonomous vehicle can be classified as higherpriority than objects traveling at lower speeds towards the autonomousvehicle. Other features can be used as well, such as object type (e.g.,vehicle, bicycle, pedestrian, etc.), object size, position, or any otherfeature described herein.

In some implementations, each object can be classified as eitherhigh-priority or low-priority. For example, the priority classificationsystem can classify each object as either high-priority or low-prioritybased on the respective state data for each object. In someimplementations, a predicted future state for each high-priority objectcan be determined before a predicted future state is determined for anylow-priority object.

In some implementations, the ratio of high-priority objects andlow-priority objects can be determined based at least in part on avelocity of the autonomous vehicle. For example, in someimplementations, in order to reduce the overall latency for determininga motion plan at higher speeds, fewer objects may be classified ashigh-priority than at lower speeds. For example, one or more thresholdsor ranges can be used to determine a ratio of high-priority objects tolow-priority objects based on a velocity of the autonomous vehicle. Eachobject can then be classified as either high-priority or low-prioritybased on the ratio of high-priority objects to low-priority objects.

According to another aspect of the present disclosure, the priorityclassification systems and methods described herein can include orleverage one or more machine-learned models that assist in classifyingthe objects. As an example, in some implementations, the priorityclassification system can include a machine-learned object priorityclassifier to classify each object perceived by the autonomous vehicle.In some implementations, the machine-learned object priority classifiercan classify each object as either high-priority or low-priority.

According to yet another aspect of the present disclosure, themachine-learned models included in or employed by the priorityclassification systems described herein can be trained using log datacollected during actual operation of autonomous vehicles on travelways(e.g., roadways). For example, the log data can include sensor dataand/or state data for various objects perceived by an autonomous vehicle(e.g., the perception system of an autonomous vehicle) and also theresulting future state for each object that occurred subsequent and/orcontemporaneous to collection of the sensor data and/or generation ofthe state data. Thus, the log data can include a large number ofreal-world examples of objects paired with the data collected and/orgenerated by the autonomous vehicle (e.g., sensor data, map data,perception data, etc.) contemporaneous to such perception, such aswhether the object became more or less likely to interact with theautonomous vehicle in the resulting future state of the object. Trainingthe machine-learned models on such real-world log data can enable themachine-learned models to determine object classifications which bettermirror or mimic real-world object behavior.

According to additional aspects of the present disclosure, theprediction system can determine a predicted future state for each objectbased at least in part on the priority classification for each object.For example, the order at which the prediction system determines thepredicted future state for each object can be based at least in part onthe priority classification assigned to the objects. For example, insome implementations, predicted future states for higher priorityobjects can be determined before predicted future states for lowerpriority objects are determined. In some implementations, the predictedfuture state for each object classified as high-priority can bedetermined before the predicted future state is determined for anyobjects classified as low-priority. In some implementations, thepredicted future state for each object can be determined based upon anobject's relative priority as compared to each other object. Forexample, each object perceived by the autonomous vehicle can be assigneda relative priority rank (e.g., for Y objects, a rank of 1 to Y), and apredicted future state can be determined based on the priority rank ofthe objects.

The prediction system can predict the future locations of the objectsbased at least in part on perception information (e.g., the state datafor each object) received from the perception system, map data, sensordata, and/or any other data that describes the past and/or current stateof the objects, the autonomous vehicle, the surrounding environment,and/or relationship(s) therebetween. For example, the prediction systemcan estimate the future motion of actors or other objects over aplanning horizon which corresponds to the period of time for which amotion plan for the autonomous vehicle is generated. In someimplementations, the prediction system can attach probabilitylikelihoods to each predicted motion or other future location of theobjects.

In some implementations, the prediction system can receive the priorityclassification for each respective object perceived by the autonomousvehicle and the respective state data for each object for a plurality ofconsecutive time frames. For example, the perception system can providestate data for a plurality of objects at a plurality of consecutive timeframes, and the priority classification system can provide a respectivepriority classification for each object for each of the plurality ofconsecutive time frames.

In some implementations, upon receiving the priority classifications andrespective state data for the plurality of objects for a current (i.e.,most recently obtained) time frame, the prediction system can determinea predicted future state for each object classified as high-priority forthe current time frame. As used herein, the terms “current” or “mostrecently obtained” when used in reference to a time frame refers to thetime frame most recently provided to a particular system (e.g.,perception system, prediction system). For example, using state data forthe current time frame, a predicted future location can be determinedfor each high-priority object. Once a predicted future state has beendetermined for each object classified as high-priority for the currenttimeframe, the predicted future states for each object classified ashigh-priority can then be provided to the motion planning system. Thus,as soon as a predicted future state has been determined for eachhigh-priority object, a motion plan can be determined for the autonomousvehicle.

Further, in some implementations, after the prediction system hasprovided the predicted future state for each object classified ashigh-priority to the motion planning system, the prediction system candetermine a predicted future state for the current timeframe for eachobject classified as low-priority. Thus, each object perceived by anautonomous vehicle can have a predicted future state determined by theprediction system for each time frame.

Additionally, in some implementations, the prediction system can providea predicted future state for a previous sequential timeframe for eachobject classified as low-priority to the motion planning systemconcurrently with the predicted future states for each object classifiedas high-priority for the current timeframe. For example, as soon as apredicted future state for the current time frame has been determinedfor each high-priority object, the predicted future states for thecurrent time frame for each high-priority object can be provided to themotion planning system along with the predicted future state for theprevious sequential time frame for each low-priority object. Forexample, a predicted future state for each object classified aslow-priority can be determined by the prediction system by selecting,obtaining, or otherwise using a predicted future state for eachlow-priority object from a previous sequential time frame. Thus, ratherthan waiting until a predicted future state has been determined for eachobject perceived by an autonomous vehicle in a current time frame, afull set of predicted future states comprising current predicted futurestates for high-priority objects and previous sequential predictedfuture states for low-priority objects can be provided to the motionplanning system as soon as the prediction system determines a predictedfuture state for all high-priority objects. This can reduce the overalllatency for determining a motion plan for a vehicle, thereby reducingthe response time for an autonomous vehicle and increasing passengersafety.

In some implementations, the prediction system can include alow-fidelity prediction system and a high-fidelity prediction system. Asused herein, the terms “low-fidelity” and “high-fidelity” refer to arelative computational intensity of the prediction system or algorithmsused by the respective prediction system. For example, in someimplementations, a high-fidelity prediction system can include orotherwise leverage one or more machine-learned models in order topredict a future location for each object. For example, in someimplementations, the prediction system can be a goal-oriented predictionsystem that generates one or more potential goals, selects one or moreof the most likely potential goals, and develops one or moretrajectories by which the object can achieve the one or more selectedgoals. For example, the prediction system can include a scenariogeneration system that generates and/or scores the one or more goals foran object and a scenario development system that determines the one ormore trajectories by which the object can achieve the goals. In someimplementations, the prediction system can include a machine-learnedgoal-scoring model, a machine-learned trajectory development model,and/or other machine-learned models. In some implementations, ahigh-fidelity prediction system can be used to determine a predictedfuture state for objects classified as high-priority.

In some implementations, a low-fidelity prediction system can includeone or more state forward integration models. For example, alow-fidelity prediction system can predict a future state for an objectby forward integrating a current state. For example, a low-fidelityprediction system can use a current position, a current velocity, and acurrent heading of an object to determine a predicted future locationfor the object at a future time period. In some implementations, alow-fidelity prediction system can be used to determine a predictedfuture state for objects classified as low-priority.

In this way, the systems and methods according to example aspects of thepresent disclosure can allow for determining a priority classificationfor objects perceived by an autonomous vehicle. In particular, byapplying one or more heuristic processes and/or using machine-learnedmodels, the systems and methods of the present disclosure can determinea respective priority classification for each object perceived by anautonomous vehicle. The order at which a predicted future state isdetermined for each object can then be determined based at least uponthe respective priority classification for each object. The ability toclassify objects according to a respective priority can allow forcomputational resources to be focused on higher-priority objects.

As such, one technical effect and benefit of the present disclosure isreduced latency for determining a predicted future location for higherpriority objects which are more likely to impact a motion plan for anautonomous vehicle than low-priority objects. In particular, the presentdisclosure provides techniques that enable a computing system todetermine a motion plan for an autonomous vehicle as soon as a predictedfuture location for all high-priority objects has been determined. Thus,the present disclosure can allow for a reduction in the time requiredfor an autonomous vehicle to perceive an object and determine a motionplan in response to the object. Further, the present disclosure canallow for higher fidelity prediction systems to be used to determinepredicted future locations for higher priority objects, and lowerfidelity prediction systems to be used to determine predicted futurelocations for lower priority objects. This can allow for more efficientuse of computing resources on board an autonomous vehicle.

The present disclosure also provides additional technical effects andbenefits, including, for example, enhancing passenger safety. Forexample, the systems and methods according to example aspects of thepresent disclosure can allow for reduced reaction times for determininga motion plan in response to an object perceived by the autonomousvehicle. This can allow an autonomous vehicle to come to a stop morequickly, navigate around the object, or otherwise respond to the objectmore quickly, thereby reducing the likelihood of an autonomous vehiclecolliding with the object.

With reference now to the FIGS., example aspects of the presentdisclosure will be discussed in further detail. FIG. 1 depicts a blockdiagram of an example autonomous vehicle 10 according to example aspectsof the present disclosure. The autonomous vehicle 10 can include one ormore sensors 101, a vehicle computing system 102, and one or morevehicle controls 107. The vehicle computing system 102 can assist incontrolling the autonomous vehicle 10. In particular, the vehiclecomputing system 102 can receive sensor data from the one or moresensors 101, attempt to comprehend the surrounding environment byperforming various processing techniques on data collected by thesensors 101, and generate an appropriate motion path through suchsurrounding environment. The vehicle computing system 102 can controlthe one or more vehicle controls 107 to operate the autonomous vehicle10 according to the motion path.

The vehicle computing system 102 can include one or more processors 112and a memory 114. The one or more processors 112 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a computing device, a microcontroller, etc.) and can be oneprocessor or a plurality of processors that are operatively connected.The memory 114 can include one or more non-transitory computer-readablestorage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices,magnetic disks, etc., and combinations thereof. The memory 114 can storedata 116 and instructions 118 which can be executed by the processor 112to cause vehicle computing system 102 to perform operations.

As illustrated in FIG. 1 , the vehicle computing system 102 can includea perception system 103, a prediction system 104, and a motion planningsystem 105 that cooperate to perceive the surrounding environment of theautonomous vehicle 10 and determine a motion plan for controlling themotion of the autonomous vehicle 10 accordingly.

In particular, in some implementations, the perception system 103 canreceive sensor data from the one or more sensors 101 that are coupled toor otherwise included within the autonomous vehicle 10. As examples, theone or more sensors 101 can include a Light Detection and Ranging(LIDAR) system, a Radio Detection and Ranging (RADAR) system, one ormore cameras (e.g., visible spectrum cameras, infrared cameras, etc.),and/or other sensors. The sensor data can include information thatdescribes the location of objects within the surrounding environment ofthe autonomous vehicle 10.

As one example, for a LIDAR system, the sensor data can include thelocation (e.g., in three-dimensional space relative to the LIDAR system)of a number of points that correspond to objects that have reflected aranging laser. For example, a LIDAR system can measure distances bymeasuring the Time of Flight (TOF) that it takes a short laser pulse totravel from the sensor to an object and back, calculating the distancefrom the known speed of light.

As another example, for a RADAR system, the sensor data can include thelocation (e.g., in three-dimensional space relative to the RADAR system)of a number of points that correspond to objects that have reflected aranging radio wave. For example, radio waves (e.g., pulsed orcontinuous) transmitted by the RADAR system can reflect off an objectand return to a receiver of the RADAR system, giving information aboutthe object's location and speed. Thus, a RADAR system can provide usefulinformation about the current speed of an object.

As yet another example, for one or more cameras, various processingtechniques (e.g., range imaging techniques such as, for example,structure from motion, structured light, stereo triangulation, and/orother techniques) can be performed to identify the location (e.g., inthree-dimensional space relative to the one or more cameras) of a numberof points that correspond to objects that are depicted in imagerycaptured by the one or more cameras. Other sensor systems can identifythe location of points that correspond to objects as well.

As another example, the one or more sensors 101 can include apositioning system. The positioning system can determine a currentposition of the vehicle 10. The positioning system can be any device orcircuitry for analyzing the position of the vehicle 10. For example, thepositioning system can determine position by using one or more ofinertial sensors, a satellite positioning system, based on IP address,by using triangulation and/or proximity to network access points orother network components (e.g., cellular towers, WiFi access points,etc.) and/or other suitable techniques. The position of the vehicle 10can be used by various systems of the vehicle computing system 102.

Thus, the one or more sensors 101 can be used to collect sensor datathat includes information that describes the location (e.g., inthree-dimensional space relative to the autonomous vehicle 10) of pointsthat correspond to objects within the surrounding environment of theautonomous vehicle 10. In some implementations, the sensors 101 can belocated at various different locations on the autonomous vehicle 10. Asan example, in some implementations, one or more cameras and/or LIDARsensors can be located in a pod or other structure that is mounted on aroof of the autonomous vehicle 10 while one or more RADAR sensors can belocated in or behind the front and/or rear bumper(s) or body panel(s) ofthe autonomous vehicle 10. As another example, camera(s) can be locatedat the front or rear bumper(s) of the vehicle 10 as well. Otherlocations can be used as well.

In addition to the sensor data, the perception system 103 can retrieveor otherwise obtain map data 126 that provides detailed informationabout the surrounding environment of the autonomous vehicle 10. The mapdata 126 can provide information regarding: the identity and location ofdifferent travelways (e.g., roadways), road segments, buildings, orother items or objects (e.g., lampposts, crosswalks, curbing, etc.); thelocation and directions of traffic lanes (e.g., the location anddirection of a parking lane, a turning lane, a bicycle lane, or otherlanes within a particular roadway or other travelway); traffic controldata (e.g., the location and instructions of signage, traffic lights, orother traffic control devices); and/or any other map data that providesinformation that assists the vehicle computing system 102 incomprehending and perceiving its surrounding environment and itsrelationship thereto.

The perception system 103 can identify one or more objects that areproximate to the autonomous vehicle 10 based on sensor data receivedfrom the one or more sensors 101 and/or the map data 126. In particular,in some implementations, the perception system 103 can determine, foreach object, state data that describes a current state of such object(also referred to as features of the object). As examples, the statedata for each object can describe an estimate of the object's: currentlocation (also referred to as position); current speed (also referred toas velocity); current acceleration; current heading; currentorientation; size/shape/footprint (e.g., as represented by a boundingshape such as a bounding polygon or polyhedron); type/class (e.g.,vehicle versus pedestrian versus bicycle versus other); yaw rate;distance from the autonomous vehicle; minimum path to interaction withthe autonomous vehicle; minimum time duration to interaction with theautonomous vehicle; and/or other state information.

In some implementations, the perception system 103 can determine statedata for each object over a number of iterations. In particular, theperception system 103 can update the state data for each object at eachiteration. Thus, the perception system 103 can detect and track objects(e.g., vehicles) that are proximate to the autonomous vehicle 10 overtime.

The prediction system 104 can receive the state data from the perceptionsystem 103 and predict one or more future locations for each objectbased on such state data. For example, the prediction system 104 canpredict where each object will be located within the next 5 seconds, 10seconds, 20 seconds, etc. As one example, an object can be predicted toadhere to its current trajectory according to its current speed. Asanother example, other, more sophisticated prediction techniques ormodeling can be used.

The motion planning system 105 can determine a motion plan for theautonomous vehicle 10 based at least in part on the predicted one ormore future locations for the object and/or the state data for theobject provided by the perception system 103. Stated differently, giveninformation about the current locations of objects and/or predictedfuture locations of proximate objects, the motion planning system 105can determine a motion plan for the autonomous vehicle 10 that bestnavigates the autonomous vehicle 10 relative to the objects at suchlocations. In some implementations, the motion planning system 105 candetermine the motion plan for the autonomous vehicle using one or moreadjusted vehicle parameters, as described herein.

In some implementations, the motion planning system 105 can evaluate oneor more cost functions and/or one or more reward functions for each ofone or more candidate motion plans for the autonomous vehicle 10. Forexample, the cost function(s) can describe a cost (e.g., over time) ofadhering to a particular candidate motion plan while the rewardfunction(s) can describe a reward for adhering to the particularcandidate motion plan. For example, the reward can be of opposite signto the cost.

Thus, given information about the current locations and/or predictedfuture locations of objects, the motion planning system 105 candetermine a total cost (e.g., a sum of the cost(s) and/or reward(s)provided by the cost function(s) and/or reward function(s)) of adheringto a particular candidate pathway. The motion planning system 105 canselect or determine a motion plan for the autonomous vehicle 10 based atleast in part on the cost function(s) and the reward function(s). Forexample, the motion plan that minimizes the total cost can be selectedor otherwise determined. The motion planning system 105 can provide theselected motion plan to a vehicle controller 106 that controls one ormore vehicle controls 107 (e.g., actuators or other devices that controlgas flow, steering, braking, etc.) to execute the selected motion plan.

According to example aspects of the present disclosure, the vehiclecomputing system 102 can also include a priority classification system150 configured to classify one or more objects perceived by theautonomous vehicle 10. For example, in some implementations, thepriority classification system 150 can receive state data descriptive ofone or more objects perceived by the autonomous vehicle 10 from theperception system 103. The priority classification system 150 can thenclassify each object based at least in part on the respective state datafor each object.

For example, in some implementations, the priority classification foreach object can be based on an object's position, velocity, and/orheading. For example, objects that are closer to the autonomous vehiclecan be given a higher priority classification. Similarly, objects thatare traveling in a direction towards the autonomous vehicle and/ortowards a position at which the autonomous vehicle will be at aforthcoming time period can be given a higher priority classification.In some implementations, objects that are traveling at higher speeds,such as objects traveling at higher speeds towards the autonomousvehicle, can be given a higher priority classification than objectstraveling at lower speeds.

In some implementations, the priority classification can be based on alikelihood that an object will interact with the autonomous vehicle orotherwise be of importance to determining a motion plan for theautonomous vehicle. For example, objects traveling in an oppositedirection as the autonomous vehicle can be given a lower priorityclassification than objects traveling in a direction that will interactwith a motion path of the autonomous vehicle.

In some implementations, the priority classification can be based on anobject type. For example, in some implementations, pedestrians can beassigned a higher priority than other objects, such as a static (i.e.,non-moving) vehicle. Similarly, other object types and/or classes can beused to determine a priority classification for each object.

In some implementations, the priority classification for each object canbe based on a minimum path to interaction with the autonomous vehicle ora minimum time duration to interaction with the autonomous vehicle. Forexample, a minimum path to interaction with the autonomous vehicle cancorrespond to a distance along one or more travelways that the objectwould have to traverse in order to interact with the object. Thus, forexample, a vehicle traveling along a highway in an opposite direction asthe autonomous vehicle may need to exit the highway, turn around,re-enter the highway, and overtake the autonomous vehicle in order tointeract with the autonomous vehicle. In such a case, the vehicle islikely to have a long minimum path to interaction and/or minimum timeduration to interaction with the autonomous vehicle. Conversely, avehicle approaching an intersection at a perpendicular path of travel tothe autonomous vehicle is likely to have a shorter minimum path tointeraction and/or minimum time duration to interaction with theautonomous vehicle. In such a case, the vehicle approaching theintersection can be given a higher priority classification than thevehicle traveling in the opposite direction.

In some implementations, the priority classification system 150 canclassify each object as high-priority or low-priority. For example, eachobject can be classified according to a binary classification in whicheach object is either a high-priority or low-priority object. Forexample, objects which have a minimum path to interaction and/or minimumtime duration to interaction with an autonomous vehicle that is lessthan a threshold can be classified as high-priority objects. Similarly,objects which have a minimum path to interaction and/or minimum timeduration to interaction that exceeds the threshold can be classified aslow-priority objects. In some implementations, objects of a particulartype (e.g., pedestrians) can always be classified as high-priorityobjects. In some implementations, objects which have been determined tobe unlikely to interact with the autonomous vehicle or the determinationof a motion plan for the autonomous vehicle can be classified aslow-priority objects.

In some implementations, the priority classification system 150 canclassify each object relative to other objects perceived by theautonomous vehicle. For example, in some implementations, each objectcan be assigned a relative priority in relation to each other objectperceived by the autonomous vehicle. For example, each object can beassigned a relative priority rank based on the respective priority ofthe object. For example, if an autonomous vehicle perceives Y objectswithin a surrounding environment of the autonomous vehicle, each objectcan be assigned a relative rank on a scale of 1 to Y. In this way, eachobject can be assigned a priority classification relative to each otherobject perceived by the autonomous vehicle.

In some implementations, the priority classification system 150 canclassify each object based on a velocity of the autonomous vehicle. Insome implementations, a ratio of high-priority objects to low-priorityobjects can be determined based on a velocity of the vehicle. Forexample, at higher velocities, it may be preferable to limit the numberof high-priority objects in order to reduce and/or minimize the numberof high-priority objects for which the prediction system 104 mustdetermine a future predicted state in a current time frame in order toreduce and/or minimize a latency for determining a motion plan inresponse to such objects. In such a case, fewer objects may beclassified as high-priority objects than at lower velocities.

In some implementations, the ratio of high-priority objects tolow-priority objects can be determined based on one or more thresholdvelocities. For example, for a first velocity range of 1 to X, a ratioof 1 high-priority object to Y low-priority objects can be used, whereasfor a second velocity range of X to 2X, a ratio of 1 high priorityobject to 2Y low-priority objects can be used. In other implementations,other pre-determined ratios can be used. In some implementations, eachobject can be classified as either high-priority or low-priority suchthat the ratio of high-priority objects to low-priority objectsgenerally conforms to the pre-determined ratio (i.e., the ratio ofhigh-priority to low-priority objects is within a threshold variance ofthe pre-determined ratio).

In some implementations, a machine-learned model can be used todetermine the priority classification for each object based on therespective state data for each object. For example, in someimplementations, the machine-learned model can be configured to classifyeach object as either high-priority or low-priority and provide thepriority classification for each object to the prediction system 104. Insome implementations, the respective state data for each object can beinput into the machine-learned model, and data indicative of arespective priority classification for the object can be received as anoutput of the machine-learned model.

In some implementations, the machine-learned model can be trained basedat least in part on training data that comprises annotated vehicle datalogs that were previously collected during previous autonomous vehicledriving sessions. For example, vehicle data logs can be recorded duringone or more autonomous vehicle driving sessions, which can include statedata for objects perceived by the autonomous vehicle. In someimplementations, the vehicle data logs can be annotated by a humanreviewer in order to help train the machine-learned model. For example,in some implementations, objects can be labeled either high-priority orlow-priority. The machine-learned model can then be trained to determinea priority classification for objects based on the training data.

According to example aspects of the present disclosure, the vehiclecomputing system 102 can determine a predicted future state for eachobject based at least in part on the priority classification for eachobject. For example, the priority classification system 150 can beconfigured to provide the priority classification for each objectperceived by the perception system 103 to the prediction system 104. Theprediction system 104 can then determine a predicted future state foreach object based at least in part on the priority classification foreach object.

For example, in some implementations, the order at which the computingsystem determines the predicted future state for each object can bebased at least in part on the priority classification assigned to eachobject. For example, in some implementations, predicted future statesfor all objects classified as high-priority can be determined beforepredicted future states are determined for any low-priority objects. Insome implementations, a predicted future state can be determined foreach object according to a respective priority rank for each object. Forexample, for Y objects, each object can be assigned a relative priorityrank of 1 to Y, and a predicted future state for each object can bedetermined based on the relative priority rank for each object (i.e.,starting with 1 and ending with Y).

As will be discussed in greater detail with respect to FIG. 5 , in someimplementations, a future location prediction system can be selectedbased at least in part on the priority classification for each object.For example, in some implementations, a low-fidelity prediction systemcan be used to determine a predicted future state for low-priorityobjects, and a high-fidelity prediction system can be used to determinea predicted future state for high-priority objects.

Each of the perception system 103, the prediction system 104, the motionplanning system 105, the vehicle controller 106, and the priorityclassification system 150 can include computer logic utilized to providedesired functionality. In some implementations, each of the perceptionsystem 103, the prediction system 104, the motion planning system 105,the vehicle controller 106, and the priority classification system 150can be implemented in hardware, firmware, and/or software controlling ageneral purpose processor. For example, in some implementations, each ofthe perception system 103, the prediction system 104, the motionplanning system 105, the vehicle controller 106, and the priorityclassification system 150 includes program files stored on a storagedevice, loaded into a memory and executed by one or more processors. Inother implementations, each of the perception system 103, the predictionsystem 104, the motion planning system 105, the vehicle controller 106,and the priority classification system 150 includes one or more sets ofcomputer-executable instructions that are stored in a tangiblecomputer-readable storage medium such as RAM hard disk or optical ormagnetic media.

Referring now to FIG. 2 , a block diagram depicting an exampleperception system 103 according to example aspects of the presentdisclosure is shown. Elements that are the same or similar to thoseshown in FIG. 1 are referred to with the same reference numerals.

As shown, in some implementations, the priority classification system150 can be implemented as a subpart of the perception system 103. Forexample, the perception system 103 can receive sensor data from one ormore sensors 101 (as shown in FIG. 1 ) and map data 126. The perceptionsystem 103 can generate state data for each object perceived by theautonomous vehicle 10, perform a priority classification for each objectusing the priority classification system 150, and provide the state dataand respective priority classification for each object to the predictionsystem 104.

Referring now to FIG. 3 , a block diagram depicting an exampleprediction system 104 according to example aspects of the presentdisclosure is shown. Elements that are the same or similar to thoseshown in FIGS. 1 and 2 are referred to with the same reference numerals.

As shown, in some implementations, the priority classification system150 can be implemented as a subpart of the prediction system 104. Forexample, the perception system 103 can receive sensor data from one ormore sensors 101 (as shown in FIG. 1 ) and map data 126. The perceptionsystem 103 can then provide the state data indicative of one or moreobjects to the prediction system 104. The prediction system 104 can thendetermine a priority classification for each object using the priorityclassification system 150, and determine a predicted future state foreach object based at least in part on the priority classification foreach object. The prediction system 104 and then provide the predictedfuture states for each object to the motion planning system 105.

Thus, as shown in FIGS. 1-3 , the priority classification system 150 canbe implemented as a stand-alone priority classification system 150, oras a sub system of either a perception system 103 or a prediction system104.

Referring now to FIG. 4 , a diagram of an example object predictionprocess according to example aspects of the present disclosure is shown.As represented in FIG. 4 , in some implementations, a vehicle computingsystem can iteratively determine a motion plan using data obtained in aplurality of consecutive time frames. For example, each of theperception, prediction, and motion planning systems illustrated in FIGS.1-3 can concurrently perform analysis on data from a plurality ofconsecutive time frames. As an example, for a time frame N, theperception system can receive sensor data for the time frame N; theperception system can concurrently generate and provide state data tothe prediction system for one or more objects perceived by theperception system for a time frame N−1; the prediction system canconcurrently determine a predicted future state for each objectperceived by the perception system for a time frame N−2; and a motionplanning system can concurrently determine a motion plan for theautonomous vehicle using predicted future states for a time frame N−3.In a subsequent time frame N+1, each of the perception, prediction, andmotion planning systems can receive and perform a respective analysis ofdata received from an upstream system, resulting in the motion planningsystem determining a motion plan using predicted future states for atime frame N−2. In this way, a motion plan for the autonomous vehiclecan be iteratively determined using data from each of a plurality ofconsecutive time frames.

For example, as shown in FIG. 4 , block 410 represents analysis by aperception system for data from a frame N. As shown, the perceptionsystem's analysis of data for frame N can include a plurality of objects411A-J. Each of the objects can have associated state data descriptiveof the object generated by the perception system. For example, for eachobject 411A-J, the perception system can generate state data describinga position, velocity, acceleration, heading, size, type, yaw rate, orother state data descriptive of the object as described herein.

As represented by the arrow from block 410 to block 430, the state datadescriptive of the objects 411A-J generated by the perception system forframe N can be provided to the prediction system once the perceptionsystem has completed its analysis.

According to example aspects of the present disclosure, however, theprediction system can also receive a priority classification for eachobject. For example, in some implementations, each object can beclassified as either high-priority (“HP”) or low-priority (“LP”). Asdescribed herein, the priority classification for each object can bedetermined based on the respective state data for each object. Further,in some implementations, the priority classification can be determinedby a machine-learned model.

Thus, as represented by block 430, the prediction system can receive therespective priority classifications for each object as well as therespective state data describing each object from the perception system.The perception system can then determine a predicted future state foreach object based at least in part on the respective priorityclassification for each object. For example, in some implementations,the prediction system can first determine a predicted future state foreach object classified as high-priority. For example, as shown in FIG. 4, the prediction system can first determine a predicted future state forHP objects 431A-D. Stated differently, the prediction system candetermine a predicted future state for each object classified ashigh-priority based at least in part on the state data obtained for themost recent time frame (Frame N).

According to additional example aspects of the present disclosure, oncethe prediction system has determined a predicted future state for eachobject classified as high-priority, the prediction system can providethe predicted future state for each object classified as high-priorityfor the current timeframe to the motion planning system. For example, asshown by the arrow from the dashed block 440 to the block 450, once theprediction system has determined a predicted future state for eachhigh-priority object HP 431A-D, the prediction system can provide thepredicted future states for the objects HP 431A-D to the motion planningsystem. In this way, the motion planning system can begin determining amotion plan in an advanced fashion (e.g., “ahead of schedule”).

According to additional example aspects of the present disclosure, oncethe prediction system has determined a predicted future state for eachobject classified as high-priority, the prediction system can determinea predicted future state for each object identified as low-priority. Forexample, after the prediction system has provided the high-priorityobjects HP 431A-D to the motion planning system, the prediction systemcan determine a predicted future state for each low priority object LP431E-J. In this way, each object perceived in a particular frame (e.g.,frame N) can have a predicted future state determined by the predictionsystem.

In some implementations, the prediction system can further be configuredto provide a predicted future state for the previous sequentialtimeframe for each object classified as low-priority to the motionplanning system concurrently with the predicted future state for eachobject classified as high-priority for the current timeframe. Stateddifferently, in some implementations, a predicted future state for alow-priority object can be determined by selecting, obtaining, orotherwise determining a predicted future state for the object based onstate data obtained for a previous sequential time frame.

For example, as shown by block 420, the prediction system can havepreviously determined a predicted future state for objects 421A-J,including high-priority objects HP 421A-D and low-priority objects LP421E-J. For example, as the perception system generated state data forobjects 411A-J for time frame N in block 410, the prediction systemcould concurrently determine predicted future states for high-priorityobjects HP 421A-D and low-priority objects LP 421E-J for time frame N−1in block 420. Further, as an example, each high-priority object HP421A-D can respectively correspond to each high-priority object HP431A-D for the time frame N−1, whereas each low-priority object LP421E-J can respectively correspond to each low-priority object LP 431E-Jfor the time frame N−1.

Thus, as represented by the arrow from block 440 to block 450, when theprediction system provides the predicted future states for high-priorityobjects HP 431A-D to the motion planning system, the prediction systemcan be configured to concurrently provide a previously determinedpredicted future state for each low-priority object (i.e., LP 431E-J)for the previous sequential time frame (i.e., LP 421E-J). In this way, afull set of predicted future states comprising the predicted futurestates for all high-priority objects (HP 431A-D) for a current timeframe and a previously determined predicted future state for alllow-priority objects (LP 421E-J) for a previous sequential time framecan be concurrently provided to a motion planning system as soon as theprediction system has determined a predicted future state for eachobject classified as high-priority (HP 431A-D).

An advantage provided by the object prediction process depicted in FIG.4 is that the time required to determine a motion plan for an autonomousvehicle can be reduced. For example, for a vehicle autonomy system suchas the sequential vehicle autonomy system described herein, the motionplanning system can receive a predicted future state for each objectmuch sooner, thereby allowing a motion plan to be determined ahead ofschedule. Further, the reduction in time for the prediction system todetermine a predicted future state for each object can correspond to theratio of high-priority objects to low-priority objects. For example, asdepicted in FIG. 4 , the prediction system would only need to determinea predicted future state for high-priority objects HP 431A-D (i.e., 4out of 10 objects) for time frame N before providing the predictedfuture states for each object 431A-D and 421E-J to the motion planningsystem, allowing for a reduction of approximately 60% of the requiredprocessing time.

Further, because low-priority objects can be classified as such based ontheir negligible impact on a motion plan, using a predicted future statefor a low-priority object from a previous sequential time frame canallow for a net increase in passenger and autonomous vehicle safety. Forexample, low-priority objects, such as objects positioned far away froman autonomous vehicle and traveling away from the autonomous vehicle,may be unlikely to impact the motion plan for the autonomous vehicle.However, high-priority objects, such as objects travelling towards theautonomous vehicle or positioned near the autonomous vehicle, may bemuch more likely to impact a motion plan for the autonomous vehicle. Byallowing for such high-priority objects to be sensed by the sensors,perceived by the perception system, predicted by the prediction system,and planned for by the motion planning system in a reduced amount oftime, the autonomous vehicle can respond to high-priority objects in aquicker fashion, thereby reducing the likelihood of unsafe conditions,such as a collision.

Referring now to FIG. 5 , a block diagram of an example computing system100 according to example embodiments of the present disclosure isdepicted. Elements that are the same or similar to those in FIGS. 1-3are referred to with the same reference numerals. As shown, the examplecomputing system 100 can include a computing system 102 (e.g., a vehiclecomputing system 102 on an autonomous vehicle 10) and a machine learningcomputing system 130 that are communicatively coupled over one or morecommunication networks 180.

The computing system 102 can include one or more processor(s) 112 andmemory 114. The one or more processors 112 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 114can include one or more non-transitory computer-readable storage media,such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flashmemory devices, etc., and combinations thereof.

The memory 114 can store information that can be accessed by the one ormore processors 112. For instance, the memory 114 (e.g., one or morenon-transitory computer-readable storage mediums, memory devices) canstore data 116 that can be obtained, received, accessed, written,manipulated, created, and/or stored. The memory 114 can also storecomputer-readable instructions 118 that can be executed by the one ormore processors 112. The instructions can be software written in anysuitable programming language or can be implemented in hardware.Additionally, or alternatively, the instructions can be executed inlogically and/or virtually separate threads on processor(s) 112. Forexample, the memory 114 can store instructions that when executed by theone or more processors 112 cause the one or more processors 112 toperform any of the operations and/or functions described herein.

The computing system 102 can also include a network interface 128 usedto communicate with one or more systems or devices, including systems ordevices that are remotely located from the computing system 102. Thenetwork interface 128 can include any circuits, components, software,etc. for communicating with one or more networks (e.g., 180). In someimplementations, the network interface 128 can include, for example, oneor more of a communications controller, receiver, transceiver,transmitter, port, conductors, software and/or hardware forcommunicating data.

The computing system 102 can also include a perception system 103, aprediction system 104, a motion planning system 105, a vehiclecontroller 106, and a priority classification system 150, as describedherein. Each of the perception system 103, the prediction system 104,the motion planning system 105, the vehicle controller 106, and thepriority classification system 150 can include computer logic utilizedto provide desired functionality. In some implementations, each of theperception system 103, the prediction system 104, the motion planningsystem 105, the vehicle controller 106, and the priority classificationsystem 150 can be implemented in hardware, firmware, and/or softwarecontrolling a general purpose processor. For example, in someimplementations, each of the perception system 103, the predictionsystem 104, the motion planning system 105, the vehicle controller 106,and the priority classification system 150 can include program filesstored on a storage device, loaded into a memory and executed by one ormore processors. In other implementations, each of the perception system103, the prediction system 104, the motion planning system 105, thevehicle controller 106, and the priority classification system 150 caninclude one or more sets of computer-executable instructions that arestored in a tangible computer-readable storage medium such as RAM harddisk or optical or magnetic media.

According to an example aspect of the present disclosure, in someimplementations, the prediction system 104 can include a low-fidelityprediction system 122 and a high-fidelity prediction system 124. Forexample, in some implementations, a high-fidelity prediction system 124can include or otherwise leverage one or more machine-learned models inorder to predict a future location for each object. For example, in someimplementations, the high-fidelity prediction system 124 can be agoal-oriented prediction system that generates one or more potentialgoals, selects one or more of the most likely potential goals, anddevelops one or more trajectories by which the object can achieve theone or more selected goals. For example, the high-fidelity predictionsystem 124 can include a scenario generation system that generatesand/or scores the one or more goals for an object and a scenariodevelopment system that determines the one or more trajectories by whichthe object can achieve the goals. In some implementations, thehigh-fidelity prediction system 124 can include a machine-learnedgoal-scoring model, a machine-learned trajectory development model,and/or other machine-learned models.

In some implementations, a low-fidelity prediction system 122 caninclude one or more state forward-integration models. For example, alow-fidelity prediction system 122 can predict a future state for anobject by forward integrating a current state. For example, alow-fidelity prediction system can use a current position, a currentvelocity, and a current heading of an object to determine a predictedfuture location for the object at a future time period.

In some implementations, the computing system 102 can determine apredicted future state for each object based at least in part on thepriority classification for the object by selecting a future locationprediction system based at least in part on the priority classificationfor the object and determining the predicted future state for the objectusing the selected future location prediction system. For example, insome implementations, the low-fidelity prediction system 122 can be usedto determine a predicted future state for objects classified aslow-priority, and the high-fidelity prediction system 124 can be used todetermine a predicted future state for objects classified ashigh-priority.

An advantage provided by using a low-fidelity prediction system and ahigh-fidelity prediction system to determine the predicted future statefor each object based at least in part on the priority classificationfor each object is that computing resources can be allocated moreefficiently. For example, low-priority objects which have beenclassified as such due to their likely negligible impact on a vehiclemotion plan may not require a sophisticated prediction system, such asthe high-fidelity prediction system 124, in order to determine apredicted future state for such objects. For example, low-priorityobjects located far away from an autonomous vehicle and/or travelling ina direction away from the autonomous vehicle may have little to noimpact on the motion plan for the autonomous vehicle. As such, thegranularity provided by a goal-oriented prediction system as describedherein may provide little to no benefit over a low-fidelity predictionmodel 122, such as a simple state forward-integration model.Accordingly, by first determining a priority classification for eachobject, computational resources can be more efficiently allocated fordetermining predicted future states for each object.

According to another example aspect of the present disclosure, thepriority classification system 150 can store or include one or moremachine-learned models 120. For example, the machine-learned model 120can be or can otherwise include various machine-learned models such asdecision tree-based models, support vector machines, k-Nearest Neighbormodels, neural networks (e.g., deep neural networks), or othermulti-layer non-linear models. Example neural networks includefeed-forward neural networks, recurrent neural networks (e.g., longshort-term memory recurrent neural networks), or other forms of neuralnetworks.

In some implementations, the one or more machine-learned models 120 caninclude a machine-learned object priority classifier. For example, insome implementations, a machine-learned object priority classifier canbe configured to classify objects perceived by the perception system 103as either high-priority or low-priority. In some implementations, themachine-learned object priority classifier can be configured to rankobjects according to a respective object priority, as described herein.

In some implementations, the computing system 102 can determine apriority classification for each object using the machine-learned model120. For example, the computing system 102 can obtain data descriptiveof the machine-learned model, input the respective state data for eachobject perceived by the perception system 103 into the machine-learnedmodel 120, and receive data indicative of a respective priorityclassification for each object as an output of the machine-learnedmodel. In some implementations, the machine-learned model 120 and/or thepriority classification system 150 can be configured to provide therespective priority classification for each object to the predictionsystem 104.

In some implementations, the vehicle computing system 102 can receivethe one or more machine-learned models 120 from the machine learningcomputing system 130 over network 180 and can store the one or moremachine-learned models 120 in the memory 114. The vehicle computingsystem 102 can then use or otherwise implement the one or moremachine-learned models 120 (e.g., by processor(s) 112).

In some implementations, certain operations described herein can beperformed by a machine learning computing system 130 that is remotelylocated to the computing system 102 and in communication with thecomputing system 102 over one or more wireless networks 180 (e.g.,cellular data networks, satellite communication networks, wide areanetworks, etc.). As an example, the machine learning computing system130 can include one or more server computing devices. In the event thatplural server computing devices are used, the server computing devicescan be arranged according to a parallel computing architecture, asequential computing architecture, or combinations thereof.

The machine learning computing system 130 can include one or moreprocessors 132 and a memory 134. The one or more processors 132 can beany suitable processing device (e.g., a processor core, amicroprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.)and can be one processor or a plurality of processors that areoperatively connected. The memory 134 can include one or morenon-transitory computer-readable storage media, such as RAM, ROM,EEPROM, EPROM, one or more memory devices, flash memory devices, etc.,and combinations thereof.

The memory 134 can store information that can be accessed by the one ormore processors 132. For instance, the memory 134 (e.g., one or morenon-transitory computer-readable storage mediums, memory devices) canstore data 136 that can be obtained, received, accessed, written,manipulated, created, and/or stored. In some implementations, themachine learning computing system 130 can obtain data from one or morememory device(s) that are remote from the system 130.

The memory 134 can also store computer-readable instructions 138 thatcan be executed by the one or more processors 132. The instructions 138can be software written in any suitable programming language or can beimplemented in hardware. Additionally, or alternatively, theinstructions 138 can be executed in logically and/or virtually separatethreads on processor(s) 132. For example, the memory 134 can storeinstructions 138 that when executed by the one or more processors 132cause the one or more processors 132 to perform any of the operationsand/or functions described herein.

The machine learning computing system 130 can also include a networkinterface 164 used to communicate with one or more systems or devices,including systems or devices that are remotely located from the machinelearning computing system 130. The network interface 164 can include anycircuits, components, software, etc. for communicating with one or morenetworks (e.g., 180). In some implementations, the network interface 164can include, for example, one or more of a communications controller,receiver, transceiver, transmitter, port, conductors, software and/orhardware for communicating data.

In some implementations, the machine learning computing system 130includes one or more server computing devices. If the machine learningcomputing system 130 includes multiple server computing devices, suchserver computing devices can operate according to various computingarchitectures, including, for example, sequential computingarchitectures, parallel computing architectures, or some combinationthereof.

In addition or alternatively to the model(s) 110 at the computing system102, the machine learning computing system 130 can include one or moremachine-learned models 140. For example, the machine learned model(s)140 can be or can otherwise include various machine-learned models suchas decision tree-based models, support vector machines, k-NearestNeighbor models, neural networks (e.g., deep neural networks), or othermulti-layer non-linear models. Example neural networks includefeed-forward neural networks, recurrent neural networks (e.g., longshort-term memory recurrent neural networks), or other forms of neuralnetworks.

As an example, the machine learning computing system 130 can communicatewith the computing system 102 according to a client-server relationship.For example, the machine learning computing system 140 can implement themachine-learned model(s) 140 to provide a web service to the computingsystem 102. For example, the web service can provide object priorityclassifications to the computing system 102.

Thus, machine-learned models 110 can be located and used at thecomputing system 102 and/or machine-learned models 140 can be locatedand used at the machine learning computing system 130.

In some implementations, the machine learning computing system 130and/or the computing system 102 can train the machine-learned models 110and/or 140 through use of a model trainer 160. The model trainer 160 cantrain the machine-learned models 110 and/or 140 using one or moretraining or learning algorithms. One example training technique isbackwards propagation of errors. In some implementations, the modeltrainer 160 can perform supervised training techniques using a set oflabeled training data 162. In other implementations, the model trainer160 can perform unsupervised training techniques using a set ofunlabeled training data 162. The model trainer 160 can perform a numberof generalization techniques to improve the generalization capability ofthe models being trained. Generalization techniques include weightdecays, dropouts, or other techniques.

In particular, the model trainer 160 can train a machine-learned model110 and/or 140 based on a set of training data 162. The training data162 can include, for example, vehicle data logs from previouslycompleted autonomous vehicle driving sessions. The vehicle data logs caninclude, for example, sensor data obtained by one or more sensors of theautonomous vehicle, state data descriptive of one or more objectsperceived by the perception system 103 of the autonomous vehicle,predicted future states for objects perceived by the autonomous vehicledetermined by the prediction system 104, previous motion plansdetermined by the motion planning system 105, or other vehicle data asdescribed herein. In some implementations, the model trainer 160 can beconfigured to train the machine-learned models 110 and/or 140 bydetermining whether objects perceived by the autonomous vehicle impacteda motion plan of the autonomous vehicle.

According to another aspect of the present disclosure, the training data162 can include vehicle data logs that include object priorityclassification labels recorded by a human reviewer can be used to trainthe machine-learned model(s) 110 and/or 140. In particular, a humanreviewer can review the vehicle data logs and label object priorityclassifications for objects perceived by the perception system 103.

The model trainer 160 includes computer logic utilized to providedesired functionality, and can be implemented in hardware, firmware,and/or software controlling a general purpose processor. For example, insome implementations, the model trainer 160 includes program filesstored on a storage device, loaded into a memory and executed by one ormore processors. In other implementations, the model trainer 160includes one or more sets of computer-executable instructions that arestored in a tangible computer-readable storage medium such as RAM harddisk or optical or magnetic media.

The network(s) 180 can be any type of network or combination of networksthat allows for communication between devices. In some embodiments, thenetwork(s) 180 can include one or more of a local area network, widearea network, the Internet, secure network, cellular network, meshnetwork, peer-to-peer communication link and/or some combination thereofand can include any number of wired or wireless links. Communicationover the network(s) 180 can be accomplished, for instance, via a networkinterface using any type of protocol, protection scheme, encoding,format, packaging, etc.

FIG. 5 illustrates one example computing system 100 that can be used toimplement the present disclosure. Other computing systems can be used aswell. For example, in some implementations, the computing system 102 caninclude the model trainer 160 and the training dataset 162. In suchimplementations, the machine-learned models 110 can be both trained andused locally at the computing system 102. As another example, in someimplementations, the computing system 102 is not connected to othercomputing systems.

In addition, components illustrated and/or discussed as being includedin one of the computing systems 102 or 130 can instead be included inanother of the computing systems 102 or 130. Such configurations can beimplemented without deviating from the scope of the present disclosure.The use of computer-based systems allows for a great variety of possibleconfigurations, combinations, and divisions of tasks and functionalitybetween and among components. Computer-implemented operations can beperformed on a single component or across multiple components.Computer-implements tasks and/or operations can be performedsequentially or in parallel. Data and instructions can be stored in asingle memory device or across multiple memory devices.

Referring now to FIG. 6 , an example method (600) to determine apredicted future state for objects perceived by an autonomous vehiclebased at least in part on a priority classification for the objectsaccording to example aspects of the present disclosure is depicted.Although FIG. 6 depicts steps performed in a particular order forpurposes of illustration and discussion, the methods of the presentdisclosure are not limited to the particularly illustrated order orarrangement. The various steps of method (600) can be omitted,rearranged, combined, and/or adapted in various ways without deviatingfrom the scope of the present disclosure. The method (600) can beimplemented by a computing system, such as a computing system comprisingone or more computing devices.

At (602), the method (600) can include obtaining, by a computing system,state data descriptive of at least a current or past state of aplurality of objects that are perceived by an autonomous vehicle. Forexample, the state data can include data descriptive of one or morefeatures of an object, such as a position, a velocity, an acceleration,a heading, a yaw rate, a shape, a size, a type, a distance from theautonomous vehicle, a minimum path to interaction with the autonomousvehicle, a minimum time duration to interaction with the autonomousvehicle, any other state data described herein, or any state datadescriptive of an object perceived by an autonomous vehicle. In someimplementations, the state data can be obtained from a perception systemof the autonomous vehicle configured to generate the state data based onsensor data obtained from one or more sensors of the autonomous vehicle.

At (604) the method (600) can include determining, by the computingsystem, a priority classification for each object in the plurality ofobjects based at least in part on the respective state data for eachobject. For example, in some implementations, the priorityclassification for each object can be determined by a priorityclassification system. In some implementations, the priorityclassification can be either a high-priority or low-priorityclassification for each object. In some implementations, the priorityclassification can be a respective priority rank for each objectrelative to each other object perceived by the autonomous vehicle.

In some implementations, the priority classification for each object canbe determined by a machine-learned model. For example, at (606), themethod can include obtaining data descriptive of a machine-learnedmodel. In some implementations, data descriptive of the machine-learnedmodel can be obtained from a memory (e.g., non-transitory computerreadable media) of the computing system. In some implementations, themachine-learned model can be a machine-learned object priorityclassifier configured to classify each object as either high-priority orlow-priority based on the respective state data for each object.

At (608), the method (600) can include inputting the respective statedata for an object into the machine-learned model. For example, statedata generated by a perception system can be received by amachine-learned model of a priority classification system. Therespective state data for each object can be input into themachine-learned model in order to determine a priority classificationfor each respective object.

At (610), the method (600) can include receiving data indicative of arespective priority classification as an output of the machine-learnedmodel. For example, in some implementations, the machine-learned modelcan be a machine-learned object priority classifier configured toclassify each object as either high-priority or low-priority, and themachine-learned model can output a respective high-priority orlow-priority classification based on the respective state data for eachobject.

At (612), the method (600) can include determining, by the computingsystem, an order at which the computing system determines a predictedfuture state for each object based at least in part on the priorityclassification for each object. For example, in some implementations,each object can be classified as either high-priority or low-priority,and the order can be determined such that each high-priority object hasa predicted future state determined before a predicted future state isdetermined for any low-priority objects. In some implementations,determining an order at which the computing system determines apredicted future state for each object can be based on the priority rankassigned to each object. For example, the highest ranked object can havea predicted future state determined first, with each successive rankedobject successively determined according to the respective priority rankfor each object.

At (614), the method (600) can include determining, by the computingsystem, the predicted future state for each object based at least inpart on the determined order. For example, in some implementations, theprediction system can determine an order in which a predicted futurestate for each object classified as high-priority before determining apredicted future state for each object classified as low-priority. Insome implementations, as soon as the prediction system has determined apredicted future state for each object classified as high-priority, theprediction system can be configured to provide the predicted futurestates for each high-priority object to a motion planning system. Insome implementations, determining a predicted future state for eachobject classified as high-priority can include determining a futurestate for each high-priority object based at least in part on state dataobtained for the most recent time frame. In some implementations,determining a predicted future state for each object classified aslow-priority can include determining a predicted future state for theobject that was determined based on state data obtained for a previoussequential time frame. For example, in some implementations, theprediction system can provide a previously-determined future predictedstate for each low-priority object for a previous sequential time frameto a motion planning system at the same time that the prediction systemprovides a future predicted state for each high-priority object for acurrent time frame to the motion planning system.

In some implementations, determining a predicted future state for eachobject based at least in part on the determined order can includeselecting a future location prediction system at least in part on thepriority classification for the object and determining the predictedfuture state for the object using the selected future locationprediction system. For example, in some implementations, a predictionsystem can include a low-fidelity prediction system and a high-fidelityprediction system. In some implementations, the low-fidelity protectionsystem can be used to determine a predicted future state for eachlow-priority object, and the high-fidelity prediction system can be usedto determine a predicted future state for each high-priority object.

At (616), the method (600) can include determining a motion plan for theautonomous vehicle based at least in part on the predicted future statefor at least one of the objects. For example, a motion planning systemcan receive one or more predicted future states for one or more objectsperceived by the autonomous vehicle, and can determine a motion plan forthe autonomous vehicle based at least in part on the predicted futurestates for the one or more objects.

In this way, the systems and methods according to example aspects of thepresent disclosure can allow for determining a priority classificationfor objects perceived by an autonomous vehicle, determining a predictedfuture state for each object based on the respective priorityclassification for each object, and determining a motion plan for theautonomous vehicle based at least in part on the predicted futurestates.

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

What is claimed is:
 1. An autonomous vehicle control system comprising:one or more processors; and one or more computer-readable medium storinginstructions that when executed by the one or more processors cause thecontrol system to perform operations, the operations comprising:obtaining state data descriptive of at least a current state or a paststate of a plurality of objects that are detected by one or more sensorsof an autonomous vehicle; determining a priority classification for aparticular object relative to the plurality of objects based, at leastin part, on respective state data for the particular object; determininga predicted future state for the particular object of the plurality ofobjects based, at least in part, on the priority classification for theparticular object; and initiating a motion of the autonomous vehiclebased, at least in part, on the predicted future state of the particularobject.
 2. The control system of claim 1, wherein determining thepredicted future state for the particular object of the plurality ofobjects based, at least in part, on the priority classification for theparticular object comprises: determining an order for determining arespective predicted future state for each of the plurality of objectsbased, at least in part, on the priority classification for theparticular object; and determining the predicted future state for theparticular object of the plurality of objects based, at least in part,on the order for determining the respective predicted future state foreach of the plurality of objects.
 3. The control system of claim 2,wherein determining the predicted future state for the particular objectof the plurality of objects based, at least in part, on the order fordetermining the respective predicted future state for each of theplurality of objects, further comprises: identifying one or more objectsin a first priority category; and identifying one or more objects in asecond priority category, wherein the order for determining therespective predicted future state for each of the plurality of objectsis determined such that the respective predicted future state for theone or more objects in the first priority category are determined beforethe respective predicted future state for the one or more objects in thesecond priority category.
 4. The control system of claim 3, whereindetermining the predicted future state for the particular object of theplurality of objects based, at least in part, on the priorityclassification for the particular object, further comprises: determiningthe respective predicted future state for each of the one or moreobjects in the first priority category based, at least in part, oncurrent state data obtained for a current time frame.
 5. The controlsystem of claim 3, wherein determining the predicted future state forthe particular object of the plurality of objects based, at least inpart, on the priority classification for the particular object,comprises: determining the respective predicted future state for each ofthe one or more objects in the second priority category based, at leastin part, on past state data obtained at a previous time frame.
 6. Thecontrol system of claim 3, wherein initiating the motion of theautonomous vehicle based, at least in part, on the predicted futurestate of the particular object, comprises: determining a motion plan forthe autonomous vehicle based, at least in part, on the predicted futurestate of the particular object, wherein the motion of the autonomousvehicle is based, at least in part, on the motion plan.
 7. The controlsystem of claim 3, wherein determining the predicted future state forthe particular object of the plurality of objects based, at least inpart, on the priority classification for the particular object,comprises: identifying the respective predicted future state for each ofthe one or more objects in the first priority classification at acurrent time frame; and identifying the respective predicted futurestate for each of the one or more objects in the second priorityclassification at a subsequent time frame subsequent to the current timeframe.
 8. The control system of claim 3, wherein determining thepredicted future state for the particular object of the plurality ofobjects based, at least in part, on the priority classification for theparticular object, comprises: selecting at least one future locationprediction system of a plurality of future location prediction systemsbased, at least in part, on the priority classification for theparticular object; and determining the predicted future state for theparticular object using the at least one future location predictionsystem.
 9. The control system of claim 8, wherein the plurality offuture location prediction systems comprise a high-fidelity predictionsystem and a low-fidelity prediction system, wherein the high-fidelityprediction system can be associated with a higher computationalintensity relative to the low-fidelity prediction system.
 10. Thecontrol system of claim 9, wherein determining the predicted futurestate for the particular object of the plurality of objects based, atleast in part, on the priority classification for the particular object,comprises: determining the respective predicted future state for each ofthe one or more objects in the first priority category using thehigh-fidelity prediction system; and determining the respectivepredicted future state for each of the one or more objects in the secondpriority category using the low-fidelity prediction system.
 11. Thecontrol system of claim 9, wherein the high-fidelity prediction systemleverages one or more machine-learned models that predict a futurelocation for a respective object, the one or more machine-learned modelscomprising a machine-learned goal-scoring model that is trained togenerate and score one or more predicted goals for the respective objectand a machine-learned trajectory development model that is trained todetermine an object trajectory for at least one of the one or morepredicted goals for the respective object, wherein the future locationfor the respective object is based, at least in part, on the objecttrajectory.
 12. The control system of claim 9, wherein the low-fidelityprediction system leverages a state forward-integration model thatpredicts a future state for a respective object by forward integrating arespective current state for the respective object.
 13. An autonomousvehicle comprising: one or more processors; and one or morecomputer-readable medium storing instructions that when executed by theone or more processors cause the autonomous vehicle to performoperations, the operations comprising: obtaining state data descriptiveof at least a current state or a past state of a plurality of objectsthat are detected by one or more sensors of the autonomous vehicle;determining a priority classification for particular object relative tothe plurality of objects based, at least in part, on respective statedata for the particular object; determining a predicted future state forthe particular object of the plurality of objects based, at least inpart, on the priority classification for the particular object; andinitiating a motion of the autonomous vehicle based, at least in part,on the predicted future state of the particular object.
 14. Theautonomous vehicle of claim 13, wherein the current state or the paststate of the plurality of objects is indicative of (i) a distance fromthe autonomous vehicle, (ii) a minimum path to an interaction with theautonomous vehicle, or (iii) a minimum time duration to the interactionwith the autonomous vehicle.
 15. The autonomous vehicle of claim 14,wherein the priority classification is determined based, at least inpart, on (i) the distance from the autonomous vehicle, (ii) the minimumpath to the interaction with the autonomous vehicle, or (iii) theminimum time duration to the interaction with the autonomous vehicle.16. The autonomous vehicle of claim 13, wherein the priorityclassification is determined based, at least in part, on a thresholdvelocity or a velocity range of the autonomous vehicle.
 17. Acomputer-implemented method, comprising: obtaining state datadescriptive of at least a current state or a past state of a pluralityof objects that are detected by one or more sensor of an autonomousvehicle; determining a priority classification for a particular objectrelative to the plurality of objects based, at least in part, onrespective state data for the particular object; determining a predictedfuture state for the particular object of the plurality of objectsbased, at least in part, on the priority classification for theparticular object; and initiating a motion of the autonomous vehiclebased, at least in part, on the predicted future state of the particularobject.
 18. The computer-implemented method of claim 17, whereindetermining the priority classification for the particular objectrelative to the plurality of objects based, at least in part, on therespective state data for the particular object comprises: providing thestate data as an input to a machine-learned object classifier model; andreceiving as an output of the machine-learned object classifier model,in response to receipt of the state data, a respective priorityclassification for each of the plurality of objects.
 19. Thecomputer-implemented method of claim 18, wherein the machine-learnedobject classifier model is trained based, at least in part, on trainingdata previously collected during one or more previous autonomous vehicledriving sessions.
 20. The computer-implemented method of claim 19,wherein the training data comprises previously recorded state dataindicative of one or more previously perceived objects during the one ormore previous autonomous vehicle driving sessions, wherein the trainingdata comprises one or more object labels corresponding to the one ormore previously perceived objects.